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1.  Introduction 


Physics-based  simulation  codes  regularly  focus  on  some  detailed  aspect  of  an  entire  system.  For 
example,  a  computational  structural  mechanics  simulation  of  a  kinetic  energy  penetrator 
impacting  a  tank  might  only  concentrate  on  a  portion  of  the  armor  instead  of  the  entire  vehicle. 
To  effectively  analyze  and  present  the  results,  it  is  imperative  to  put  the  simulation  in  context. 

In  addition,  the  computational  simulation  might  be  run  with  a  simplified  version  of  the  complete 
system  that  only  incorporates  the  detail  that  is  germane  to  the  calculation. 

It  can  therefore  be  beneficial  to  incorporate  Augmented  Virtuality  techniques  into  the  analysis  in 
order  to  provide  context  and  provide  additional  detail  not  always  available  from  the  virtual 
model. 

Paul  Miligram1  is  credited  with  the  term  “Augmented  Virtuality,”  which  identified  systems 
which  were  primarily  synthetic  with  some  real-world  imagery  added  to  the  virtual  objects,  as 
shown  in  figure  1 . 


- 

Mixed  Reality  (MR) 

Reality 

Augmented  Augmented 

Virtuality 

Reality  (AR)  Virtuality  (AV) 

Figure  1.  Miligram’ s  reality-virtuality  continuum. 

Commodity-based  virtual  reality  systems  have  recently  blossomed  primarily  due  to  the  increased 
computational  and  graphical  capability  of  cost-effective  PC  systems.  Combined  with  relatively 
low-cost  bright  projectors,  a  single  rear-projection  stereoscopic  system  can  be  built  for  under 
$15,000.  Impressive  systems  like  the  FlatWorld  project  at  the  University  of  Southern 
California’s  Institute  for  Creative  Technology,2  combine  these  rear-projected  screens  with 
modular  theatrical  components  and  three-dimensional  (3-D)  spatial  audio  in  order  to  deliver  a 
scalable,  reconfigurable  training  and  simulation  environment. 


1  Miligram,  P.  The  23rd  Army  Science  Conference,  Orlando,  FL,  2-5  December  2002. 

2 

Flatworld:  A  Mixed  Reality  Environment  for  Simulation  and  Training.  University  of  California  Institute  for  Creative 
Technology. 
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Systems  like  FlatWorld,  the  CAVE  (CAVE  Automatic  Virtual  Environment),  and  the  NAVE 
(Nonexpensive  Automatic  Virtual  Environment),  etc.,  have  advantages  over  head-mounted 
displays  (HMDs).  The  main  advantage  is  the  “shared  experience”  of  a  group.  This  is  a  major 
contribution  to  the  success  of  many  theme  park  attractions,  like  those  found  in  Orlando,  FL. 
Using  “mixed  reality”  techniques  to  combine  real  and  virtual  objects  with  spatial  audio  systems, 
they  demonstrate  the  ability  to  convince  a  group  to  collectively  “suspend  reality”  for  some 
purpose,  like  entertainment  or  training. 

But  the  application  of  these  mixed  reality  techniques  to  physics-based  simulations  is  not  obvious. 
Exactly  how  would  adding  sound  to  a  molecular  visualization  increase  the  insight  of  the  user? 
What  real  object  could  one  add  to  an  electromagnetic  wave  simulation  to  better  explain  the 
demonstrated  phenomenon? 


2.  The  Reality  Sandwich 


An  inherent  aspect  of  traditional  projection  screens  is  that  even  with  stereoscopic  projection,  real 
objects  can  never  be  in  front  of  virtual  objects,  they  would  block  the  projection  of  the  virtual 
image.  Using  a  unique  transparent  material  called  “TransFilm,”  however,  we  are  able  to  project 
stereoscopic  images  that  appear  to  be  in  front  of  real  objects.  TransFilm3  has  the  special 
property  of  preserving  the  polarization  of  light,  a  property  not  shared  by  other  transparent  screen 
materials.  By  adding  a  traditional  rear  projection  system,  we  can  create  a  “reality  sandwich,” 
consisting  of  Virtual-Real- Virtual  objects  sandwiched  together  as  shown  in  figure  2.  Naturally, 
additional  configurations  are  now  possible  (Real- Virtual-Real,  etc.). 

The  transparent  screen  approach  differs  from  other  efforts  like  the  “Being  There”  project4  where 
synthetic  textures  are  projected  onto  real  objects.  With  a  transparent  screen,  virtual  objects  are 
not  visually  limited  to  the  surface  of  the  object. 

We  have  observed  that  one  must  pay  great  attention  to  proper  lighting.  While  real  objects  must 
be  brightly  lit,  ambient  light  on  the  TransFilm  ruins  the  apparent  “holographic”  effect. 

The  effect  of  semi-transparent  objects  that  appear  to  hover  in  mid  air  is  not  new.  In  fact,  there  is 
a  well-known  technique  called  “Pepper’s  Ghost”  that  is  used  in  such  places  as  the  Haunted 
Mansion  at  Walt  Disney  World  in  Orlando,  FL.  The  name  is  taken  from  Professor  John  Henry 
Pepper,  who  amazed  audiences  in  the  late  1800s  with  the  illusion  of  a  transparent  ghost  that 
seemed  to  hover  in  mid  air  above  the  stage.  It’s  a  simple  technique  that  consists  of  nothing  more 
than  the  reflection  on  an  angled  piece  of  glass. 


3  . 

TransFilm.  Laser  Magic  Productions,  Los  Angeles,  CA;  http://www.laser-magic.com/transscreen.html. 

4Kok-Lim,  L.;  Welch,  G.;  Lastra,  A.;  Fuchs,  H.  Life-sized  Projector-Based  Dioramas.  Proceedings  of  ACM  Symposium  on 
Virtual  Reality,  Software  and  Technology  (VRST  2001),  Banft  Calgary,  Canada,  November  2001. 
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Figure  2.  Augmented  Virtuality  prototype  system  at  ARL. 


3.  Data  Soniflcation 


But  visual  representation  is  only  part  of  the  puzzle.  For  a  total  immersion  into  simulation,  the 
Scientific  Visualization  (Sci  Vis)  team  is  using  established  “data  sonification”  techniques  to 
place  3-D  spatial  sounds  in  the  environment.  This  allows  for  more  dimensions  of  data  to  be 
simultaneously  experienced  than  would  be  possible  with  visual  techniques  alone. 

According  to  H.  G.  Kaper5  of  the  Scientific  Sonification  Project  at  Argonne  National  Laboratory, 
“Sound,  alone  or  in  combination  with  visual  imaging  techniques,  offers  a  powerful  means  of 
transmitting  information.  It  can  significantly  increase  the  bandwidth  of  the  human/computer 
interface.” 

For  example,  if  one  were  to  launch  a  data  probe  into  a  volumetric  dataset,  the  probe  could 
change  color  in  response  to  a  scalar  field.  If  the  pitch  of  a  sound  were  additionally  varied  in 
response  to  another  scalar  field,  the  user  could  intuitively  glean  the  relationship  between  the  two 
attributes. 

The  hardware  to  drive  the  sound  is  an  Audigy  2  card  from  Creative  Labs,  which  provides  6.1 
surround- sound  capability.  A  commodity  audio/visual  (A/V)  receiver  outfitted  with  a  powered 
subwoofer  and  omnidirectional  speakers  are  mounted  around  the  audience. 


3Kaper,  H.  G.  MCS  Division,  Argonne  National  Laboratory:  Argonne,  IL;  http://www.unix.mcs.anl.gov/~kaper 
/sonification. 
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Additionally,  we  have  constructed  a  viewing  platform  and  installed  a  device  that  isolates 
low-frequency  audio  signals  and  transforms  them  into  tactile  vibrations  with  a  powerful  piston. 
This  allows  the  third  sense  of  “feel”  to  convey  additional  data.  Incidentally,  the  viewing 
platform  performs  the  additional  function  of  eliminating  the  “hot  spot”  from  the  audience’s  view 
created  by  the  TransFilm’s  rear  projectors. 


4.  Passive  Stereoscopic  Graphics 


In  addition  to  being  less  expensive  than  active  systems,  passive  stereo  has  the  advantage  being 
more  scalable  in  a  distributed  environment  because  multiple  screens  do  not  need  to  sync  the 
video  signal. 

Currently,  we  are  using  dual  InFocus  LP650  Digital  Micromirror  Devices  (DLP)  projectors 
driven  by  a  single  NVIDIA  GeForce  4  dual  display  graphics  card. 

For  passive  stereo,  polarizing  filters  are  placed  over  the  projectors  at  right  angles  as  shown  in 
figure  3.  Glasses  with  the  same  filter  material  are  worn  to  pass  the  proper  projected  image  to  the 
correct  eye. 


Figure  3.  Dual  DLP  projectors  with  polarizing  filters. 


Because  the  projectors  that  the  Sci  Vis  team  are  using  require  a  15-ft  “throw”  distance,  they 
project  the  image  onto  an  ordinary,  but  good  quality,  bathroom  mirror.  The  image  is  then 
directed  to  the  screen.  Both  the  TransFilm  and  the  traditional  rear-projection  screen  must 
preserve  the  polarization  of  the  light. 
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5.  Nothing  but  Net 


A  major  problem  with  systems  composed  of  many  individual  parts  is  control.  To  assist  in  this 
area,  great  care  has  been  taken  to  ensure  that  all  parts  are  controllable  via  software.  The 
projectors  have  a  network  interface  and  are  controlled  via  a  custom  Python  layer.  Other 
electrical  devices  are  controlled  via  an  X10  interface  that  sends  control  signals  across  the 
standard  electrical  wiring.  The  Sci  Vis  team  has  developed  a  Python  interface  to  the 
X10  controller  so  that  applications  can  set  the  necessary  lighting  conditions  without  user 
intervention.  Finally,  hardware  services  that  are  only  found  on  certain  PCs,  like  the  sound  and 
speech  on  the  node  with  the  Audigy  2  card,  are  made  available  to  other  nodes  via  transmission 
control  protocol/Internet  protocol  (TCP/IP)  socket-based  servers  and  listed  in  a  common  XML 
system  description. 


6.  Software  Integration 


From  the  outset,  an  integrated  system  was  envisioned.  A  PC-based  Augmented  Virtuality 
scientific  data  center  is  of  little  use  without  a  scalable  software  interface. 

Data  from  physics-based  simulations  come  into  the  system  via  a  common  data  model  and  format. 
We  are  using  the  “extensible  Data  Model  and  Format”  (XDMF)  developed  at  the  U.S.  Army 
Research  Laboratory  (ARL)  to  provide  a  common  data  interface  for  the  development  of  scalable 
codes  and  tools  on  high-performance  computing  systems.  XDMF  combines  XML  and  HDF5 
from  the  National  Center  for  Supercomputing  Applications  (NCSA)  with  a  custom 
heterogeneous  shared  memory  system  to  provide  a  common  method  to  exchange  data  during 
runtime  or  for  post-processing. 

The  visualization  toolkit  (VTK)  is  an  enormous  collection  of  visualization  algorithms  and 
graphics  facilities.  VTK  has  an  XDMF  reader  which  allows  us  to  import  data  from  many 
different  codes  into  the  VTK  visualization  pipeline. 

The  Sci  Vis  team  is  using  a  platform-independent  software  layer  called  OpenAL  to  place  various 
sounds  in  3-D  space.  OpenAL  allows  for  the  placement  of  multiple  objects  in  3-D  space  and  for 
effects  such  as  reverb  and  Doppler  shift.  An  additional  layer,  EAX,  allows  for  additional  effects. 

The  Sci  Vis  team  has  added  an  additional,  object-oriented  convenience  layer  on  top  of  OpenAL 
in  order  to  easily  place  sounds  in  our  environment  and  synchronize  their  movement  with  the 
graphical  output. 
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Additionally,  the  team  has  developed  another  object-oriented  convenience  layer  in  order  to 
interface  with  the  Microsoft  “Speech”  facility  to  provide  both  a  text  to  speech  and  speech 
recognition  capability.  This  allows  the  entire  complex  system  to  be  controlled  via  simple  voice 
commands.  For  example,  “Front  Projectors  On”  and  “Ceiling  Lights  Off’  are  two  of  the  more 
useful  commands  of  the  system. 

All  of  the  various  layers  have  been  “wrapped”  to  provide  an  interface  to  the  Python  scripting 
language.  Thus,  all  applications  are  developed  at  a  high  level  of  abstraction  without  sacrificing 
much  perfonnance. 


7.  Current  Prototype  System 


The  Sci  Vis  team  currently  has  a  prototype  system  consisting  of  three  2.6-GHz  Pentium  4 
systems  with  NVIDIA  GeForce  4  dual  display  graphics  cards.  One  of  the  systems  incorporates 
an  Audigy  2  Platinum  sound  card  connected  to  a  Yamaha  A/V  receiver/switcher.  For  speech 
recognition,  a  dynamic  microphone  is  also  connected  to  the  Audigy  2.  The  Yamaha  drives  six 
Mirage  OmniSat  speakers  and  a  150-W  powered  subwoofer. 

The  Sci  Vis  team  constructed  a  frame  made  from  electrical  conduit  structural  steel  known  as 
“channel  strut.”  The  frame  holds  a  traditional  rear  projection  screen  and  a  TransFilm  screen. 
Additionally,  the  frames  support  two  InFocus  LP650  projectors  for  each  screen. 

An  8-  x  8-ft  viewing  platform  sits  3  ft  from  the  frame  and  is  outfitted  with  the  “shaker” 
(commercially  available  as  the  “ButtKicker  II”  from  The  Guitammer  Company  Inc.),  which  is 
driven  by  a  1000-W  amplifier.  Because  the  Sci  Vis  team  built  the  prototype  on  the  second  floor 
of  their  building,  they  have  certainly  “got  the  attention”  of  their  downstairs  neighbors  when  the 
sound  system  is  at  full  volume. 

The  Sci  Vis  team  used  the  system  to  explore  electromagnetic  pulses  over  various  geometries  of 
interest  and  to  explore  penetrator-armor  interaction.  The  sound  systems  allow  the  team  to  map 
scalar  fields  to  sounds  and  to  probe  the  datasets  both  visually  and  acoustically.  The  TransFilm 
allows  for  the  team  to  place  representative  objects  in  the  middle  of  computed  isosurfaces  and 
volume  visualizations.* 


The  Sci  Vis  team  used  the  system  to  explore  electromagnetic  pulses  over  various  geometries  of  interest,  to  explore 
penetrator-armor  interaction,  and  guntube  dynamics.  The  sound  system  allows  them  to  map  scalar  fields  to  sounds  and  probe  the 
datasets  both  visually  and  acoustically.  For  example,  the  XM1002,  a  training  round  for  the  Ml  A1  Abrams  Main  Battle  Tank, 
was  studied  in  the  system  through  sonification  of  scalar  values  representing  guntube  pressure.  Areas  of  high  pressure  within  the 
guntube  were  represented  by  louder,  high-pitch  sounds,  while  low-pressure  areas  were  represented  with  softer,  low-pitch  sounds. 
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8.  Future  Configuration 


Each  of  the  frames,  with  projectors,  mirrors,  sound,  and  computer  system  is  known  as  a  mixed 
reality  pod  or  “MrPod.”  From  these  scalable  units,  the  Sci  Vis  team  has  begun  to  construct  a 
multipod  facility.  Small  pods  with  8-  x  6-ft  screens  and  large  pods  with  12-  x  9-ft  screens  will 
be  combined  to  provide  a  scalable  data  center  that  can  be  reconfigured  as  needed. 

The  team’s  current  tracking  needs  are  modest.  The  accuracy  of  the  magnetic  systems  like  those 
from  Polhemus  Inc.  is  not  necessary  in  most  of  our  applications.  The  Sci  Vis  team  is  currently 
exploring  the  use  of  infrared  image-based  tracking  for  objects  inside  the  pods. 


9.  Conclusion 


Physics-based  simulations  provide  important  insight  into  the  behavior  of  complex  systems.  A 
unique  application  of  Augmented  Reality  techniques  to  the  analysis  of  these  simulations  provides 
an  intuitive  understanding  of  the  complex  relationships  between  computed  values.  In  addition, 
the  use  of  commodity  components  and  a  little  imagination  makes  for  a  cost-effective  solution  for 
presenting  complex  phenomena. 
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